Add signed MSRT campaign DAGs and TP-aware EXL3 cartridge artifacts - #41

- Fix regex to match BF16 tensor names (*.weight) not just EXL3 (*.rank0.trellis)
- Fix expert filter to handle 'hot96' string key from recipe
- Add base_dir.mkdir before writing files
- Restore corrupted imports and remove duplicate definitions

Co-authored-by: Claude <noreply@anthropic.com>

quantize_tiles returns raw Viterbi indices (n_tiles, 256). EXL3 checkpoints
store PACKED indices (n_tiles, K*16) — compressed via ext.pack_trellis.
The vLLM loader validates this at exl3.py:2099-2102.

Fixed quantize_trellis_packed to:
1. Collect raw 256-width indices from qtf
2. Call ext.pack_trellis to compress to K*16 packed format
3. Return packed tensor with correct (k//16, n//16, K*16) shape

Also added ext parameter threading through encode_expert_msrt and
rescaled_trellis_quantize. Falls back to raw indices when ext is None
(for testing without the CUDA extension).

Co-authored-by: Claude <noreply@anthropic.com>

Since vLLM supports only 1 LoRA per request, this tool combines individual
MSRT stage files into single adapter files:
- cart_k3like: K1trsc for all experts (K3-equivalent, 3bpw)
- cart_k3k4like: K1trsc (all) + K2trsc (96 hot) = matches SIQ 160K3+96K4

Co-authored-by: Claude <noreply@anthropic.com>

Weight-level MSE (3 layers, 10 experts each, gate/up/down projections):

| Config | MSE | vs K3 | vs K4 |
|--------|-----|-------|-------|
| K3 only | 2.718e-02 | 1.00× | 3.73× |
| K4 only | 7.284e-03 | 0.268× | 1.00× |
| MSRT K2+K1trsc | 2.908e-02 | 1.07× | 3.99× (K3-equiv, 7% worse) |
| MSRT K2+K1+K2trsc | 1.995e-03 | 0.073× | 0.274× (3.6× BETTER than K4!) |

MSRT at 4bpw (K2+K1trsc+K2trsc) is 3.6× better than native K4 at the same
bitrate. This confirms v50/v52 PoC results on the real Fruit model.

Co-authored-by: Claude <noreply@anthropic.com>

…apes, scalar mcg)

- suh: (input_size,) float16 (was float32, wrong shape)
- svh: (output_size,) float16 (was float32, wrong shape)
- mcg: scalar () int32 (was (1,) int32)

These match the SIQ model's checkpoint format that vLLM's EXL3 loader expects.
Without this fix, the K2 base checkpoint fails validation with:
  ValueError: Invalid EXL3 MoE tensors for expert=0, projection=w1

Co-authored-by: Claude <noreply@anthropic.com>

PyTorch stores Linear weights as (out_features, in_features), but EXL3
trellis expects (in//16, out//16, K*16). The encoder used k,n = w.shape
treating k=input, n=output — swapped for all projections.

This caused suh/svh and trellis dimensions to be swapped vs the SIQ
reference checkpoint:
  gate/up: suh=(512,) svh=(1024,) trellis=(32,64,K*16)  [wrong]
  should be: suh=(1024,) svh=(512,) trellis=(64,32,K*16)

Fix: transpose weight before encoding: w = w.T.contiguous()

Discovered by comparing trellis shapes against the SIQ reference model
during vLLM loading (slab geometry mismatch error).

Co-authored-by: Claude <noreply@anthropic.com>

Rewrite the cartridge encoder around the graph the products actually form,
and make every emitted fragment carry checkable provenance.

fq-cartridge/2 replaces the linear recipe: bases plus stages that name the
parent reconstruction they correct. Nine products spanning 35 nominal trellis
bpw now cost nine quantization passes emitting 14, measured at 1.83x less
trellis kernel time than encoding each product separately on real GLM-5.2
experts.

Campaign mechanics: plan/skeleton/encode/finalize; reads standard indexed HF
shards or per-layer shards without loading a whole shard; work addressed as
(layer, 32-expert block), claimed with an O_EXCL lock and committed as one
atomic unit so a resumed block can never pin a residual to a parent it only
recomputed in memory; --devices runs one worker per GPU over disjoint blocks.
TILE_BATCH=128 is the measured tiling optimum at both model scales.

Provenance: each shard ships a signed fq-attestation/1 line naming the sha256
of every expert's contiguous byte range, the encoder bundle including the
compiled extension, the determinism scope, the quant args, and the exact
parent shard digest a residual corrects. Shard payloads carry no timestamp, so
re-encoding inside the declared scope reproduces them byte for byte. finalize
re-hashes every fragment before publishing and refuses two signers, two
encoder builds, unbound chains or shards the recipe does not describe.

Consumers pin a key: fq_combine_cartridges verifies the signed assembly plan,
requires the campaign identity to match every fragment, checks each chain edge,
re-derives the runtime constants instead of copying them, validates tensor
geometry, and narrows a product to chosen experts from the signed plan before
reading any payload.

Fixes found while building this: a hardlinked config.json made both bases claim
one K and mutated the skeleton; finalize wiped a base's commit markers; a
zero-RMS residual shipped an all-zero trellis that decodes to a nonzero
codebook value; save_file's dtype ordering meant no expert occupied one byte
range, so expert_sha256 could not mean what it says.

Evidence: 13 GPU tests on an RTX 5090 with the real EXL3 kernels, including a
published campaign decoded end to end through ext.reconstruct; 420 local tests.

A cartridge is only valid over the reconstruction it was encoded against. The
signed plan already pinned the base by manifest digest and named the base block
bytes its first stage corrects; --base makes the consumer compare both with the
checkpoint that will actually be loaded, and checks the base fragment's
signature under the same pinned key.

Also pin the emitted tensor-key set in a test. A scripted edit had dropped
suh_/svh_ from stage shards: still valid safetensors, still decodable trellis
indices, reconstructs nothing. The combiner's component check caught it; now a
unit test states the runtime contract directly.

fq-attestation/1 required expert_sha256 unconditionally, so the skeleton
shards' per-tensor lines were signed but invalid under the schema they claimed,
and fq_verify would have rejected them. The schema now carries two profiles
(expert fragments require expert_sha256; a fragment declaring kind:"skeleton"
requires tensor_sha256), which leaves every previously valid document valid, and
fq_verify enforces the same split. A test validates every line a real campaign
emits through both paths.

The procedure had never been run whole. Running it on the Fruit SIQ proxy (11
layers, 256 experts, 88 blocks, one RTX 5090) took 67 minutes and produced the
numbers in §3.5: 0.54% commit overhead, an idempotent resume, finalize verifying
792 expert fragments, nine published assemblies, and two products whose decode
through the runtime's own ext.reconstruct lands 14.5x and 52.7x below their K2
base, at 1.40x the block-mean MSE the deepest stage attested.

It also found two things reading the code would not have: a two-launcher campaign
that finalize correctly refused for mixed signer identity, and stage shards
missing their Hadamard sign vectors.

Two of the nine products only sell a +1-bit upgrade to a consumer who already
installed a 3 or 4 bpw tier, and they are the two most expensive nodes in the
graph: a K1 residual on a K1 residual. Measured on all 88 blocks of the Fruit
rehearsal and on real GLM-5.2 experts, that narrow-step path is 9.0% (K2 family)
and 6.8% (K3 family) worse than fetching the wider residual at the same bitrate:
88/88 blocks, sigma 0.0002, and the proxy agrees with GLM to three decimals.

recipes/glm52-k2k3-lean.json drops those two stages. Both graphs then encoded one
real 32-expert GLM block back to back on the same clean RTX 5090:

  glm52-k2k3-dag   9 passes  1106.6 s  11.5276 s/matrix  ->  186.9 GPU-h
  glm52-k2k3-lean  7 passes   785.0 s   8.1775 s/matrix  ->  132.6 GPU-h

54.3 GPU-hours and $58 of compute, 7.2 h of fleet wall time and 186 GB of
storage, for seven products each better than the one it replaces. The measured
lean/full ratio (0.7093) lands 0.2% from what the per-K table predicted
(0.7108), so the table can price a graph change before anyone rents a card.

Also fixed, found by re-running the rehearsal's own finalize: finalize was not
idempotent. Publishing a base hardlinks the skeleton into it, and the
stale-shard check then rejected those names, so a preemption during a
multi-terabyte verify pass would have reported 'remove them or encode into a
fresh --out' on a finished campaign. Finalize now accepts the skeleton names it
publishes itself while still refusing shards from any other block layout.

Measured, not budgeted: finalize verifies at 908 MB/s; encode is byte-identical
across runs (new test crosses a wall-clock second); the per-node bookkeeping the
loop was suspected of is 0.06% of a matrix.

The recipe decision rested on 88 proxy blocks and three experts of one
projection in one layer. Swept real GLM-5.2 instead: layers 3, 10, 19, 30, 40,
50, 60, 70, six experts each, all three projections, both residual families.
168 of 168 comparisons favour the wider residual; sigma 0.00022 (K2 family) and
0.00044 (K3); the three projections agree to four decimals. The ordering is a
property of greedy residual trellis coding, not of a layer or a model.

Section 4 was missing the two commands an operator actually types: the
hf download that stages a window, and the deletion that reclaims it. Both are
now generated from window.json so they cannot drift from the plan, the delete
list is built from shard names only, and the runbook states what must survive:
config.json and the index. Verified by running finalize against a source
directory holding nothing else - it completed and published every product.
$RECIPE is now anchored at the checkout instead of being relative to whatever
directory the operator happens to be in.

…ding

Adversarial review of the operator path found five defects that would each have
surfaced only after money was spent. All are fixed with tests or with a command
that was actually executed.

Encoder:
- campaign_lock: one flock per campaign directory, held for the launcher's
  lifetime and released by the kernel if it dies. A launcher clears leftover
  per-block claims before forking, which was safe only if no other launcher was
  alive; without this lock a second launcher deleted live claims and both fleets
  encoded the same blocks, then the first to finish unlinked the second's claim.
  encode, skeleton and finalize are now mutually exclusive; spawned workers
  inherit the lock and skip it.
- the sentinel binds signer_pubkey, and bind_encoder_identity records or enforces
  one encoder build. A wrong --sign-key or a rebuilt exllamav3 used to encode
  happily and fail at finalize, after the fleet had been paid; the rehearsal lost
  82 GPU-minutes that way. Both now fail in seconds.
- plan emits skeleton_only_shards. For GLM-5.2 one shard holds only embeddings,
  lm_head and the dense layers, so it appears in no layer's shard list: staging
  strictly per layer left it absent and finalize refused to publish a base a
  whole campaign later.

Procedure:
- tools/msrt_campaign.sh is the whole loop as one checked driver, with DRY_RUN.
  Verified by running it end to end on the Fruit proxy: two windows, staging,
  skeleton, 48 blocks at 0.3399 s/matrix, last-use retirement keeping exactly the
  skeleton-only and later-window shards, then finalize.
- staging moved off the HF cache. Verified on the box: snapshot entries are
  76-byte symlinks into blobs/, so deleting a window freed nothing and the source
  would have accumulated 1.5067 TB against a 1.6 TB volume. --local-dir writes
  real files; identity is passed explicitly since the tree is no longer a
  snapshot path.
- window geometry measured from the pinned index and the hub tree listing: the
  ten windows are 198.6/166.2/160.7/166.2/171.6/166.2/160.8/166.2/166.2/85.8 GB,
  not a uniform 150, and 20 shards straddle a boundary. Retaining them costs
  5.3 GB of disk and saves 107.2 GB of WAN. Peak local bytes: lean 1.266 TB, dag
  1.439 TB, so 1.6 TB runs either graph and the old advice to rent 2 TB was wrong.
- gates run from a fresh rental: a throwaway gate campaign, metadata staged
  first, skeleton before encode so the key exists, and --force on the timed
  full-layer run so a gate block does not make it a partial sample.
- publication goes to a staging ref and is promoted with one ref move, and the
  GPU release gate is a hard requirement with a checklist: eight GPUs left up
  through finalize and the 3.4 h base drain cost $27, 17% of the budget.
- new §2.1 sizes the fleet: 4 GPUs is the $149.98 optimum, 8 costs $2.05 more
  and saves 17.6 h. Budget re-itemised to $160 centre, $200 cap.

Also recipes/fruit-k2k3-lean.json, so the rehearsal exercises the graph the
campaign will run.

The procedure itself had never been run as one command. tools/msrt_campaign.sh
now has, end to end on the Fruit proxy with the lean recipe: two windows, rolling
--local-dir staging, last-use retirement that kept exactly the skeleton-only and
later-window shards, the release gate, finalize, both products, and a decode that
came out bit-identical to the earlier hand-run campaign under a different key.
Recorded as §3.6.

Review also found the roofline factor was one dag-derived number applied to both
graphs. It is a function of the K1 share, which differs: dag 0.7235 -> 1.0617,
lean 0.6109 -> 1.0372. Lean's rental centre is therefore 137.5 GPU-h / $136, not
140.8 / $139. The card-range rows are relabelled as duration multipliers, since
x0.90 duration is 11% more throughput, not 10%.

Other claim-discipline fixes from the same review: the finalize rate is 9.32 GB
(8.68 GiB) in 10.263 s = 908 MB/s and its extrapolation to a cold 1.146 TB tree is
labelled unmeasured; the throughput-to-hide-I/O figures now divide by the rental
wall, not the local one, and say the first window and final drain overlap nothing;
the budget cap moves to $220 because $200 left no margin over its own worst
column; the preemption claim is labelled conditional on three unmeasured provider
behaviours; Xet is disabled for source staging, where each shard is fetched once
and the chunk cache is unbounded against the same volume; and the gate commands
are anchored at $REPO instead of the working directory.

Fixed two latent breakages in the GPU-only parity test, found by running it:
13 passed on the RTX 5090.

The campaign lock stopped two launchers, but not the case it cannot see: kill a
launcher and its workers keep running, the kernel drops the campaign lock, a new
launcher starts and -- under the old scheme -- deleted the orphans' O_EXCL claim
files as 'stale' and handed the same blocks out again.

Ownership is now an flock on the claim file instead of the file's existence. The
kernel releases it however the owner dies, so a crashed worker leaves nothing to
clean up and no launcher has to decide whether someone else's claim is stale.
The launcher's clearing pass is gone. Lock files are deliberately never unlinked:
unlinking a path another process already opened would flock a detached inode and
give one block two owners.

Verified with real processes: a child holding block (3,0) outlived its parent; a
fresh launcher took the campaign lock, was refused (3,0), was granted (3,1), and
got (3,0) only after the orphan exited. Also through the launcher path on GPU
against a finished campaign: 'nothing to do (8 blocks already complete)'.

…mable

Third review round, all found by asking what happens when a process dies at the
wrong moment or an operator pastes the wrong thing.

Concurrency:
- finalize held its locks only while building the expected set; the verify and
  publish pass ran unlocked. cmd_finalize now holds them through its whole body.
- a launcher's lock dies with the launcher, but its workers do not, so
  publication could overlap orphans. Workers hold .fq-campaign.workers.lock
  shared for their lifetime; finalize takes it exclusively. Both directions
  tested.
- exit 0 from every worker was treated as 'this window is finished', but a worker
  skips a block another live worker holds and still exits 0. The launcher now
  asserts every block of its own work list is committed before returning, which
  is exactly what the driver's source retirement depends on.
- bind_encoder_identity is a compare-and-set under an flock: two different builds
  could otherwise both observe 'unbound' and each record itself.
- encode_block retracts every node's markers before quantizing rather than one
  node at a time. A crash between two node commits used to leave the rewritten
  nodes with fresh markers and the rest with their old valid ones, so the block
  read as complete and resume skipped a mixture that only finalize's parent
  digest check would catch. New test crashes mid-block and asserts the block
  reads incomplete and resumes as a whole.

Procedure:
- DEVICES is required. It defaulted to cuda:0, so following the runbook literally
  on an eight-GPU node would have run one GPU while billing all eight: $1,089
  instead of $136.
- PHASE=encode/finalize split. The old script printed 'release the fleet' and
  then ran finalize itself, so the handoff it documented was impossible. The GPU
  phase now exits after checking that the key, recipe, source metadata and
  campaign are all on the volume the CPU VM will inherit.
- per-window done-markers, bound to a run identity of recipe sha, revision, block
  size and window partition. Without them a rerun after a late preemption
  re-staged every earlier window, ~1.5 TB, with eight GPUs idle; with a drifted
  partition it is refused rather than skipping work it never did.
- tools/promote_campaign.py replaces a documented command that does not exist
  ('hf repo branch merge'). It promotes a staging branch to main in ONE commit
  built from server-side copies, so no payload moves and no consumer sees a
  partial family.
- section 4 now clones and installs the tree, authenticates, defines $PROXY,
  creates the directories it writes into, and anchors the gate plan; cmd_plan
  creates --out-plan's parent.

Numbers: the lean 'products encoded separately' row was computed from the dag
product set. Its own seven products cost 12.518 s/matrix, so the graph saving is
1.58x, not 1.94x, and nominal byte sharing is 2.17x (26 vs 12 bpw), not 2.9x. The
superseded $139 / 17.6 h / 360 Mbps / 273 Mbps figures are replaced everywhere by
$136 / 17.2 h / 348 Mbps / 258 Mbps.

WINDOWS is the one operator-supplied value the tools cannot check for
themselves: each range validates individually, so 3-77 instead of 3-78 encodes 75
of 76 layers, retires the source those layers needed, and is discovered by
finalize on the CPU VM after the whole fleet has been paid. The driver now
verifies the union of the ranges equals the recipe's MoE layers with no repeats,
before staging a single byte, and names what is missing, extra or doubled.

Verified: 3-13 covers the Fruit recipe exactly; 3-12 is refused for layer 13;
'3-8 8-13' is refused for encoding 8 twice; 3-14 is refused for a layer the
recipe does not have; and the default ten-window list covers all 76 GLM layers
exactly once. Also made the recipe digest portable (sha256sum or shasum).

fq-promote-campaign is what turns a staged campaign into a published one, so it
belongs in the wheel next to the encoder and the combiner rather than only in a
checkout. Verified from a built wheel: all six console scripts respond to --help,
and both lean recipes ship in fq_data/recipes.

The user-facing surface changed: two priced recipes instead of one, a campaign
driver, a promotion tool, and enforced single-launcher/single-key semantics. The
headline numbers in both documents were still the nine-product ones (201 GPU-h,
1.332 TB, 1.83x, 2.5x); they now lead with the recommended lean graph
(133 GPU-h, 1.147 TB, 1.58x, 2.17x) and keep the nine-product figures beside it,
with a pointer to the section that prices the difference.

Fourth-round review verification found three deterministic failures in the parts
I had just changed.

1. fq-promote-campaign blacklisted operational files, so the normal campaign --
   which by then contained campaign.env, plans/gate.json and source-tree.json --
   would have been refused as 'missing' on the branch. It now enumerates what a
   campaign publishes (base/, stages/, assemblies/, campaign_summary.json) and is
   indifferent to whatever else the directory accumulates. Verified against the
   real campaign with those three files present: 1,967 publishable files, none of
   the operational ones.
2. The check compared the mutable branch and promotion resolved a SHA afterwards,
   which is the exact window the pin was supposed to close. The SHA is resolved
   first, the comparison runs against it, and every copy operation names it.
3. The handoff could not run: campaign.env did not export REPO, so the documented
    expanded to /tools/... on a
   fresh VM. It now exports REPO, DRIVER and PATH, the release gate also requires
   the driver and the CLI to be on the reattached filesystem, and the runbook uses
   $DRIVER.

Also: one honest opt-out (REHEARSAL=1) replaces ALLOW_ROOT_CAMPAIGN and covers
every survivability check, since 'the tool ships in a container image' and 'the
campaign is on the root disk' are the same question -- will this survive the
release. The runbook now states plainly that the gate cannot distinguish a
persistent volume from an instance-local disk at the same path, and lists the
provider steps only the operator can perform. The tree to check out is a tag,
msrt-campaign-ready, instead of a commit that predates its own fixes, and the
encoder revision is an explicit ENC_REV with the rehearsal's measured
encoder_sha256 and stack recorded beside it.

An empty `command -v` result was dropped from the prerequisite loop by
`${cli_path:+...}`, so a machine with no installed fq-assemble-lora passed the
gate and was told its filesystem held the CLI. The reachable path is a resume
after preemption: with every window already complete, no step invokes the tool, so
the gate is the only place its absence would be noticed before the fleet is
released. It now reports the missing tool and refuses.

REHEARSAL is a release-gate switch, so it accepts exactly 0 or 1 and refuses to
guess at 'yes' or 'false' rather than silently disabling every check. campaign.env
also no longer fabricates a PATH from `/usr/bin/false` when the tool does not
resolve.

Verified on Linux with a resumed campaign whose windows were all complete: absent
CLI and REHEARSAL=0 exits 1 naming the tool; REHEARSAL=1 warns and proceeds;
REHEARSAL=yes exits 2.